19.1.2. email.parser: Parsing email messages

Message object structures can be created in one of two ways: they can be created from whole cloth by instantiating Message objects and stringing them together via attach() and set_payload() calls, or they can be created by parsing a flat text representation of the email message.

The email package provides a standard parser that understands most email document structures, including MIME documents. You can pass the parser a string or a file object, and the parser will return to you the root Message instance of the object structure. For simple, non-MIME messages the payload of this root object will likely be a string containing the text of the message. For MIME messages, the root object will return True from its is_multipart() method, and the subparts can be accessed via the get_payload() and walk() methods.

There are actually two parser interfaces available for use, the classic Parser API and the incremental FeedParser API. The classic Parser API is fine if you have the entire text of the message in memory as a string, or if the entire message lives in a file on the file system. FeedParser is more appropriate for when you’re reading the message from a stream which might block waiting for more input (e.g. reading an email message from a socket). The FeedParser can consume and parse the message incrementally, and only returns the root object when you close the parser [1].

Note that the parser can be extended in limited ways, and of course you can implement your own parser completely from scratch. There is no magical connection between the email package’s bundled parser and the Message class, so your custom parser can create message object trees any way it finds necessary.

19.1.2.1. FeedParser API

The FeedParser, imported from the email.feedparser module, provides an API that is conducive to incremental parsing of email messages, such as would be necessary when reading the text of an email message from a source that can block (e.g. a socket). The FeedParser can of course be used to parse an email message fully contained in a string or a file, but the classic Parser API may be more convenient for such use cases. The semantics and results of the two parser APIs are identical.

The FeedParser‘s API is simple; you create an instance, feed it a bunch of text until there’s no more to feed it, then close the parser to retrieve the root message object. The FeedParser is extremely accurate when parsing standards-compliant messages, and it does a very good job of parsing non-compliant messages, providing information about how a message was deemed broken. It will populate a message object’s defects attribute with a list of any problems it found in a message. See the email.errors module for the list of defects that it can find.

Here is the API for the FeedParser:

class email.parser.FeedParser(_factory=email.message.Message, *, policy=policy.compat32)

Create a FeedParser instance. Optional _factory is a no-argument callable that will be called whenever a new message object is needed. It defaults to the email.message.Message class.

If policy is specified (it must be an instance of a policy class) use the rules it specifies to update the representation of the message. If policy is not set, use the compat32 policy, which maintains backward compatibility with the Python 3.2 version of the email package. For more information see the policy documentation.

Changed in version 3.3: Added the policy keyword.

feed(data)

Feed the FeedParser some more data. data should be a string containing one or more lines. The lines can be partial and the FeedParser will stitch such partial lines together properly. The lines in the string can have any of the common three line endings, carriage return, newline, or carriage return and newline (they can even be mixed).

close()

Closing a FeedParser completes the parsing of all previously fed data, and returns the root message object. It is undefined what happens if you feed more data to a closed FeedParser.

class email.parser.BytesFeedParser(_factory=email.message.Message)

Works exactly like FeedParser except that the input to the feed() method must be bytes and not string.

New in version 3.2.

19.1.2.2. Parser class API

The Parser class, imported from the email.parser module, provides an API that can be used to parse a message when the complete contents of the message are available in a string or file. The email.parser module also provides header-only parsers, called HeaderParser and BytesHeaderParser, which can be used if you’re only interested in the headers of the message. HeaderParser and BytesHeaderParser can be much faster in these situations, since they do not attempt to parse the message body, instead setting the payload to the raw body as a string. They have the same API as the Parser and BytesParser classes.

New in version 3.3: The BytesHeaderParser class.

class email.parser.Parser(_class=email.message.Message, *, policy=policy.compat32)

The constructor for the Parser class takes an optional argument _class. This must be a callable factory (such as a function or a class), and it is used whenever a sub-message object needs to be created. It defaults to Message (see email.message). The factory will be called without arguments.

If policy is specified (it must be an instance of a policy class) use the rules it specifies to update the representation of the message. If policy is not set, use the compat32 policy, which maintains backward compatibility with the Python 3.2 version of the email package. For more information see the policy documentation.

Changed in version 3.3: Removed the strict argument that was deprecated in 2.4. Added the policy keyword.

The other public Parser methods are:

parse(fp, headersonly=False)

Read all the data from the file-like object fp, parse the resulting text, and return the root message object. fp must support both the readline() and the read() methods on file-like objects.

The text contained in fp must be formatted as a block of RFC 2822 style headers and header continuation lines, optionally preceded by an envelope header. The header block is terminated either by the end of the data or by a blank line. Following the header block is the body of the message (which may contain MIME-encoded subparts).

Optional headersonly is a flag specifying whether to stop parsing after reading the headers or not. The default is False, meaning it parses the entire contents of the file.

parsestr(text, headersonly=False)

Similar to the parse() method, except it takes a string object instead of a file-like object. Calling this method on a string is exactly equivalent to wrapping text in a StringIO instance first and calling parse().

Optional headersonly is as with the parse() method.

class email.parser.BytesParser(_class=email.message.Message, *, policy=policy.compat32)

This class is exactly parallel to Parser, but handles bytes input. The _class and strict arguments are interpreted in the same way as for the Parser constructor.

If policy is specified (it must be an instance of a policy class) use the rules it specifies to update the representation of the message. If policy is not set, use the compat32 policy, which maintains backward compatibility with the Python 3.2 version of the email package. For more information see the policy documentation.

Changed in version 3.3: Removed the strict argument. Added the policy keyword.

parse(fp, headersonly=False)

Read all the data from the binary file-like object fp, parse the resulting bytes, and return the message object. fp must support both the readline() and the read() methods on file-like objects.

The bytes contained in fp must be formatted as a block of RFC 2822 style headers and header continuation lines, optionally preceded by an envelope header. The header block is terminated either by the end of the data or by a blank line. Following the header block is the body of the message (which may contain MIME-encoded subparts, including subparts with a Content-Transfer-Encoding of 8bit.

Optional headersonly is a flag specifying whether to stop parsing after reading the headers or not. The default is False, meaning it parses the entire contents of the file.

parsebytes(bytes, headersonly=False)

Similar to the parse() method, except it takes a byte string object instead of a file-like object. Calling this method on a byte string is exactly equivalent to wrapping text in a BytesIO instance first and calling parse().

Optional headersonly is as with the parse() method.

New in version 3.2.

Since creating a message object structure from a string or a file object is such a common task, four functions are provided as a convenience. They are available in the top-level email package namespace.

email.message_from_string(s, _class=email.message.Message, *, policy=policy.compat32)

Return a message object structure from a string. This is exactly equivalent to Parser().parsestr(s). _class and policy are interpreted as with the Parser class constructor.

Changed in version 3.3: Removed the strict argument. Added the policy keyword.

email.message_from_bytes(s, _class=email.message.Message, *, policy=policy.compat32)

Return a message object structure from a byte string. This is exactly equivalent to BytesParser().parsebytes(s). Optional _class and strict are interpreted as with the Parser class constructor.

New in version 3.2.

Changed in version 3.3: Removed the strict argument. Added the policy keyword.

email.message_from_file(fp, _class=email.message.Message, *, policy=policy.compat32)

Return a message object structure tree from an open file object. This is exactly equivalent to Parser().parse(fp). _class and policy are interpreted as with the Parser class constructor.

Changed in version Removed: the strict argument. Added the policy keyword.

email.message_from_binary_file(fp, _class=email.message.Message, *, policy=policy.compat32)

Return a message object structure tree from an open binary file object. This is exactly equivalent to BytesParser().parse(fp). _class and policy are interpreted as with the Parser class constructor.

New in version 3.2.

Changed in version 3.3: Removed the strict argument. Added the policy keyword.

Here’s an example of how you might use this at an interactive Python prompt:

>>> import email
>>> msg = email.message_from_string(myString)

19.1.2.3. Additional notes

Here are some notes on the parsing semantics:

  • Most non-multipart type messages are parsed as a single message object with a string payload. These objects will return False for is_multipart(). Their get_payload() method will return a string object.
  • All multipart type messages will be parsed as a container message object with a list of sub-message objects for their payload. The outer container message will return True for is_multipart() and their get_payload() method will return the list of Message subparts.
  • Most messages with a content type of message/* (e.g. message/delivery-status and message/rfc822) will also be parsed as container object containing a list payload of length 1. Their is_multipart() method will return True. The single element in the list payload will be a sub-message object.
  • Some non-standards compliant messages may not be internally consistent about their multipart-edness. Such messages may have a Content-Type header of type multipart, but their is_multipart() method may return False. If such messages were parsed with the FeedParser, they will have an instance of the MultipartInvariantViolationDefect class in their defects attribute list. See email.errors for details.

Footnotes

[1]As of email package version 3.0, introduced in Python 2.4, the classic Parser was re-implemented in terms of the FeedParser, so the semantics and results are identical between the two parsers.